{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[View the runnable example on GitHub](https://github.com/intel-analytics/BigDL/tree/main/python/nano/tutorial/notebook/inference/openvino/accelerate_inference_openvino_gpu.ipynb)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Accelerate Inference on Intel GPUs Using OpenVINO"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use `InferenceOptimizer.trace(..., accelerator='openvino', device='GPU')` to enable the OpenVINO acceleration for inference on Intel GPUs, both integrated and discrete ones. BigDL-Nano also supports quantization with OpenVINO accelerator on Intel GPUs by `InferenceOptimizer.quantize(..., accelerator='openvino', device='GPU', precision='fp16'/'int8')`. It only takes a few lines."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "nbsphinx": "hidden"
   },
   "source": [
    "To apply OpenVINO acceleration, you need to install BigDL-Nano first:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "nbsphinx": "hidden"
   },
   "outputs": [],
   "source": [
    "# for pytorch users\n",
    "!pip install --pre --upgrade bigdl-nano[pytorch,inference] # install the nightly-built version\n",
    "\n",
    "!source bigdl-nano-init"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "nbsphinx": "hidden"
   },
   "outputs": [],
   "source": [
    "# for tensorflow users\n",
    "!pip install --pre --upgrade bigdl-nano[tensorflow,inference] # install the nightly-built version\n",
    "\n",
    "!source bigdl-nano-init"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbsphinx": "hidden"
   },
   "source": [
    "> 📝 **Note**\n",
    "> \n",
    "> We recommend to run the commands above, especially `source bigdl-nano-init` before jupyter kernel is started, or some of the optimizations may not take effect."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbsphinx": "hidden"
   },
   "source": [
    "> ⚠️ **Warning**\n",
    ">\n",
    "> Errors may occur when using `InferenceOptimizer.trace(..., accelerator='openvino')` API in CentOS, becuase the latest version of `openvino-dev` is not supported in CentOS."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before starting this guide, below codes can help you search the available Intel GPU devices on you machine, and you can inference on any one of them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from openvino.runtime import Core\n",
    "core = Core()\n",
    "print(core.available_devices)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function returns a list of available devices:\n",
    "\n",
    "|output|corresponding GPU device(s)|\n",
    "|-|-|\n",
    "|`GPU`|alias for `GPU.0`, integrated GPU|\n",
    "|`GPU.X`|enumeration of GPUs, `X` - id of the GPU device|\n",
    "|`GPU.X.Y`|specific tile in a multi-tile architecture, `X` - id of the GPU device, `Y` - id of the tile within device `X`|\n",
    "\n",
    "For more information around the device naming convention of openvino, you can refer to this [page](https://docs.openvino.ai/latest/openvino_docs_OV_UG_supported_plugins_GPU.html#device-naming-convention)."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## PyTorch example"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a [ResNet-18 model](https://pytorch.org/vision/main/models/generated/torchvision.models.resnet18.html) pretrained on ImageNet dataset as an example. Note that you don't have to transfer the model to GPU and set it to evaluation mode since `InferenceOptimizer` will handle these automatically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "nbsphinx": "hidden"
   },
   "outputs": [],
   "source": [
    "# Define the finetune function\n",
    "import torch\n",
    "from torchvision import transforms\n",
    "from torchvision.datasets import OxfordIIITPet\n",
    "from torch.utils.data.dataloader import DataLoader\n",
    "from torchvision.models import resnet18\n",
    "from bigdl.nano.pytorch import Trainer\n",
    "from torchmetrics.classification import MulticlassAccuracy\n",
    "\n",
    "\n",
    "def finetune_pet_dataset(model_ft):\n",
    "\n",
    "    train_transform = transforms.Compose([transforms.Resize(256),\n",
    "                                          transforms.RandomCrop(224),\n",
    "                                          transforms.RandomHorizontalFlip(),\n",
    "                                          transforms.ColorJitter(brightness=.5, hue=.3),\n",
    "                                          transforms.ToTensor(),\n",
    "                                          transforms.Normalize([0.485, 0.456, 0.406],\n",
    "                                                               [0.229, 0.224, 0.225])])\n",
    "    val_transform = transforms.Compose([transforms.Resize(256),\n",
    "                                        transforms.CenterCrop(224),\n",
    "                                        transforms.ToTensor(),\n",
    "                                        transforms.Normalize([0.485, 0.456, 0.406],\n",
    "                                                             [0.229, 0.224, 0.225])])\n",
    "\n",
    "    # apply data augmentation to the tarin_dataset\n",
    "    train_dataset = OxfordIIITPet(root=\"/tmp/data\",\n",
    "                                  transform=train_transform,\n",
    "                                  download=True)\n",
    "    val_dataset = OxfordIIITPet(root=\"/tmp/data\",\n",
    "                                transform=val_transform)\n",
    "\n",
    "    # obtain training indices that will be used for validation\n",
    "    indices = torch.randperm(len(train_dataset))\n",
    "    val_size = len(train_dataset) // 4\n",
    "    train_dataset = torch.utils.data.Subset(train_dataset, indices[:-val_size])\n",
    "    val_dataset = torch.utils.data.Subset(val_dataset, indices[-val_size:])\n",
    "\n",
    "    # prepare data loaders\n",
    "    train_dataloader = DataLoader(train_dataset, batch_size=32)\n",
    "    val_dataloader = DataLoader(val_dataset, batch_size=32)\n",
    "\n",
    "    num_ftrs = model_ft.fc.in_features\n",
    "\n",
    "    # here the size of each output sample is set to 37.\n",
    "    model_ft.fc = torch.nn.Linear(num_ftrs, 37)\n",
    "    loss_ft = torch.nn.CrossEntropyLoss()\n",
    "    optimizer_ft = torch.optim.SGD(model_ft.parameters(), lr=0.01, momentum=0.9, weight_decay=5e-4)\n",
    "\n",
    "    # compile our model with loss function, optimizer.\n",
    "    model = Trainer.compile(model_ft, loss_ft, optimizer_ft, metrics=[MulticlassAccuracy(num_classes=37)])\n",
    "    trainer = Trainer(max_epochs=1)\n",
    "    trainer.fit(model, train_dataloaders=train_dataloader, val_dataloaders=val_dataloader)\n",
    "\n",
    "    return model, train_dataset, val_dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchvision.models import resnet18\n",
    "\n",
    "pt_model = resnet18(pretrained=True)\n",
    "_, train_dataset, val_dataset = finetune_pet_dataset(pt_model)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;_The full definition of function_ `finetune_pet_dataset` _could be found in the_ [runnable example](https://github.com/intel-analytics/BigDL/tree/main/python/nano/tutorial/notebook/inference/openvino/accelerate_inference_openvino_gpu.ipynb)."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To enable OpenVINO acceleration for your PyTorch inference pipeline on Intel GPUs, **the only change you need to made is to import BigDL-Nano** `InferenceOptimizer`**, and trace your PyTorch model to convert it into an OpenVINO accelerated module for inference, with specifying** `device='GPU'`."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 📝 **Note**\n",
    "> \n",
    "> By setting `device` to `'GPU'`, inference will be conducted on the default Intel GPU device. You can change to other devices (`'GPU.X'` / `'GPU.X.Y'`) instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from bigdl.nano.pytorch import InferenceOptimizer\n",
    "\n",
    "ov_model = InferenceOptimizer.trace(pt_model,\n",
    "                                    accelerator=\"openvino\",\n",
    "                                    input_sample=torch.rand(1, 3, 224, 224),\n",
    "                                    device='GPU')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 📝 **Note**\n",
    "> \n",
    "> `input_sample` is the parameter for OpenVINO accelerator to know the **shape** of the model input. So both the batch size and the specific values are not important to `input_sample`. If we want our test dataset consists of images with $224 \\times 224$ pixels, we could use `torch.rand(1, 3, 224, 224)` for `input_sample` here.\n",
    "> \n",
    "> Please refer to [API documentation](https://bigdl.readthedocs.io/en/latest/doc/PythonAPI/Nano/pytorch.html#bigdl.nano.pytorch.InferenceOptimizer.trace) for more information on `InferenceOptimizer.trace`."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to quantize your model by using OpenVINO Post-training Optimization Tools, you could call `InferenceOptimizer.quantize`.\n",
    "\n",
    "* For FP16 quantization:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from bigdl.nano.pytorch import InferenceOptimizer\n",
    "\n",
    "ov_model = InferenceOptimizer.quantize(pt_model,\n",
    "                                       accelerator='openvino',\n",
    "                                       input_sample=torch.rand(1, 3, 224, 224),\n",
    "                                       device='GPU',\n",
    "                                       precision='fp16')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* For INT8 quantization:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from bigdl.nano.pytorch import InferenceOptimizer\n",
    "\n",
    "ov_model = InferenceOptimizer.quantize(pt_model,\n",
    "                                       accelerator='openvino',\n",
    "                                       input_sample=torch.rand(1, 3, 224, 224),\n",
    "                                       device='GPU',\n",
    "                                       precision='int8',\n",
    "                                       calib_data=DataLoader(train_dataset, batch_size=32))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 📝 **Note**\n",
    "> \n",
    "> For INT8 quantization, we adopt the Post-training Optimization Tools provided by OpenVINO toolkit, which only supports **static** post-training quantization. So `calib_data` (calibration data) is always required when `accelerator='openvino'`. Here batch size is not important as it intends to read 100 samples. And there could be no label in calibration data.\n",
    "> \n",
    "> Please refer to [API documentation](https://bigdl.readthedocs.io/en/latest/doc/PythonAPI/Nano/pytorch.html#bigdl.nano.pytorch.InferenceOptimizer.quantize) for more information on `InferenceOptimizer.quantize`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You could then do the normal inference steps with the model optimized by OpenVINO:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with InferenceOptimizer.get_context(ov_model):\n",
    "    x = torch.rand(2, 3, 224, 224)\n",
    "    # use the optimized model here\n",
    "    y_hat = ov_model(x)\n",
    "    predictions = y_hat.argmax(dim=1)\n",
    "    print(predictions)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TensorFlow example"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take [MobileNetV2](https://www.tensorflow.org/api_docs/python/tf/keras/applications/mobilenet_v2/MobileNetV2) as an example. Note that you don't have to transfer the model to GPU at this step since `InferenceOptimizer` will handle this automatically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow.keras.applications.mobilenet_v2 import MobileNetV2\n",
    "import numpy as np\n",
    "\n",
    "tf_model = MobileNetV2(weights=None, input_shape=[40, 40, 3], classes=10)\n",
    "\n",
    "train_examples = np.random.random((100, 40, 40, 3))\n",
    "train_labels = np.random.randint(0, 10, size=(100,))\n",
    "train_dataset = tf.data.Dataset.from_tensor_slices((train_examples, train_labels))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To enable OpenVINO acceleration for your TensorFlow inference pipeline on Intel GPUs, **the only change you need to made is to import BigDL-Nano** `InferenceOptimizer`**, and trace your TensorFlow model to convert it into an OpenVINO accelerated module for inference, with specifying** `device='GPU'`."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 📝 **Note**\n",
    "> \n",
    "> By setting `device` to `'GPU'`, inference will be conducted on the default Intel GPU device. You can change to other devices (`'GPU.X'` / `'GPU.X.Y'`) instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bigdl.nano.tf.keras import InferenceOptimizer\n",
    "\n",
    "ov_model = InferenceOptimizer.trace(tf_model,\n",
    "                                    accelerator=\"openvino\",\n",
    "                                    device='GPU')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to quantize your model by using OpenVINO Post-training Optimization Tools, you could call `InferenceOptimizer.quantize`.\n",
    "\n",
    "* For FP16 quantization:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bigdl.nano.tf.keras import InferenceOptimizer\n",
    "\n",
    "ov_model = InferenceOptimizer.quantize(tf_model,\n",
    "                                       accelerator='openvino',\n",
    "                                       device='GPU',\n",
    "                                       precision='fp16')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* For INT8 quantization:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from bigdl.nano.tf.keras import InferenceOptimizer\n",
    "\n",
    "ov_model = InferenceOptimizer.quantize(tf_model,\n",
    "                                       accelerator='openvino',\n",
    "                                       device='GPU',\n",
    "                                       precision='int8',\n",
    "                                       x=train_dataset)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 📝 **Note**\n",
    "> \n",
    "> For INT8 quantization, we adopt the Post-training Optimization Tools provided by OpenVINO toolkit, which only supports **static** post-training quantization. So `x` (serves as calibration data) is always required when `accelerator='openvino'`. And there could be no label in calibration data.\n",
    "> \n",
    "> Please refer to [API documentation](https://bigdl.readthedocs.io/en/latest/doc/PythonAPI/Nano/tensorflow.html#bigdl.nano.tf.keras.InferenceOptimizer.quantize) for more information on `InferenceOptimizer.quantize`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You could then do the normal inference steps with the model optimized by OpenVINO:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = tf.random.normal(shape=(100, 40, 40, 3))\n",
    "# use the optimized model here\n",
    "y_hat = ov_model(x)\n",
    "predictions = tf.argmax(y_hat, axis=1)\n",
    "print(predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 📚 **Related Readings**\n",
    "> \n",
    "> - [How to install BigDL-Nano](https://bigdl.readthedocs.io/en/latest/doc/Nano/Overview/install.html)\n",
    "> - [How to install BigDL-Nano in Google Colab](https://bigdl.readthedocs.io/en/latest/doc/Nano/Howto/Install/install_in_colab.html)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.7.0 (default, Jun 28 2018, 13:15:42) \n[GCC 7.2.0]"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "8772eaeb16382a2d9dbb95ffcb3882976733f8dc8a0780f3e0ca9a3a7dc812c0"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
